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The National Center for Education Statistics (NCES), 
in its expansion of data reporting, should consider approaches for 
studying the curricular content of elementary and secondary 
instruction and student achievement of curricular goals. More 
specific information is needed about the curricular content of U.S. 
education. For example, data collected on the average years of high 
school mathematics studied do not indicate what proportion of 
graduates have studied algebra, trigono.netry, or calculus, or which 
skills have been mastered. This information would be useful for 
college and vocational planners, for prov;)ram evaluation and planning, 
and for fiscal accountability. Better data on curriculum and 
achievement would also allow comparisons across educational agencies 
and their programs and comparisons across time to assess trends. Data 
on instructional content and methods could include courses of study, 
enrollment, mejor skills taught, local and state guidelines, teaching 
methods, teacher qualifications, and supervisor role. Testing of 
students' mastery of instructional content should involve different 
difficulty levels, the full breadth of content, and different types 
of achievement sought by different teaching methods. Problems which 
are likely to be encountered in designing these new data collection 
activities involve: agreement among the interested parties; public 
perception of burden and expense; inappropriate comparisons; and 
diverse educational goals. (GDC) 
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NEW AREAS FOR EDUCATIONAL DATA COLLECTION: 
WHAT STUDENTS ARE TAUGHT AND WHAT THEY LEARN 



Elizabeth R. Reisner 
Policy Studies Associates 



In the next several years pressures are likely to grow for 
the federal government to expand its reporting of what elementary 
and secondary students are taught and, from the instruction to 
which they are exposed, how much they learn. Gathering accurate 
data in these two areas will present major challenges both to the 
National Center for Education Statistics, because of the tech- 
nical problems of survey and analysis design, and to its parent 
the U.S. Department of Education, because of the policy debate 
these proposals and plans will inevitably provoke. To meet these 
challenges, NCES should begin now to consider approaches for 
gathering and reporting data on (1) the carricular content of 
elementary and secondary instruction and (2) students' achieve- 
ment of the curricular goals that are set for them. 

This paper provides an overview of topics related to federal 
data collection in these areas. The first section reviews the 
current need for such data. The second section explores how data 
in these areas could actually be collected. The third section 
surveys several problems to be addressed in designing these data 
collection activities. 
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Why We Need Better Information in These Areas 



At present federal agencies provide extensive information on 
many aspect? of American education. For example, comprehensive 
information is available on what have been termed the inputs of 
the American education system; these include the characteristics 
of participating students, the fiscal resources available to ele- 
mentary and secondary education, the structure established to 
govern educational activities, and other components of the sys- 
tem. We also know something about the processes that affect edu- 
cation, including the interlocking character of governance deci- 
sions at local, state, and federal levels and the supply, assign- 
ment, and supervision of teachers and other staff. With regard 
to educational outcomes, we know how students score on tests that 
measure knowledge and critical thinking, using standard test 
items for students in very different schools and localities. 

These measures do not, however, permit us to form generali- 
zations about the curricular content of instruction or students' 
mastery of that content. For example, although we may know that 
high school students in a given state complete an average of 
three units of mathematics prior to graduating, for most states 
we do not know what proportion of graduating students take 
Algebra II, Trigonometry, or Calculus, nor do we know what mathe- 
matical skills and concepts are typically taught in each of these 
mathematics courses in a state. Without such information, we 
cannot legitimately compare the -mathematics program in one state 
with the program of another state whose graduates typically 
complete fewer units of mathematics prior to graduation. 
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similarly, without data describing students' mastery of the 
luathematics curriculum in their schools, we cannot conclude 
whether the completion of three units of mathematics in a state 
generally results in (1) mastery of a larger range of mathema- 
tical skills and concepts and (2) more thorough mastery than is 
achieved in a state whose students generally complete fewer 
units. 

It is not surprising that the federal government has avoided 
collecting detailed data in these areas. As discussed later in 
this paper, it is a particularly difficult thing to do. 

Nevertheless, the task is important, as illustrated in three 
recent reports on proposals for change in higher education 
(Bennett, 1984; National Institute of Education, 1984; and Asso- 
ciation of American Colleges, 1985). The three reports look 
critically at the content of current undergradua*" curricula and 
propose ways of upgrading it. Because of the sparsity of 
national data on undergraduate curricula, the conclusions of 
these reports are based on anecdotal evidence of what college 
curricula currently include and on small-scale surveys of course 
titles in selected colleges and universities. If these data were 
more complete ^ the authors of these reports would be able to make 
their recommendations more persuasively and indeed might find 
greater congruence among their analyses. Similarly, the lack of 
data on the content of elementary and secondary schooling 
precludes accurate analysis of the extent to which students are 
receiving instruction in skills and subject areas needed for 
successful transitions into these postsecondary programs. 
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Our lack of information on what students are taught and what 
they learn makes it difficult for us to analyze the fit between 
elementary and secondary schooling and the range of post-high 
school paths taken by young adults. For students going on to 
post secondary schooling, col lege and univers ity planners need to 
know what content and skills have jeen presented to high school 
students in order to design appropriate entry-level courses; in 
particular^ postsecondary planners need to know how the content 
of high school programs is changing, in order to adjust entry- 
level undergraduate courses accordingly. For example, if recent 
national attention to science and mathematics instruction has 
actually increased students" exposure to and mastery of skiJ.ls 
and information in these areas, entry-level undergraduate courses 
in science and mathematics should be upgraded to reflect these 
changes. For high school graduates moving directly into jobs, 
postsecondary vocational training, or military service, analysis 
of high school course content, students' mastery of it, and 
trends in content and mastery can indicate whether high schools 
are adequately preparing students for changing vocational 
requirements and whether changes at the high school level reflect 
trends in job requirements. 

These data are also needed for other policy purposes. For 
example^ they can provide a yardstick for educational agencies to 
compare their own programs and performance with those of com- 
parable agencies. This information can help them identify 
instructional areas of relative weakness and strength. Data on 
instructional content and mastery can also provide a baseline for 
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agencies to assess changes in their programs and student perfor- 
mance over time. This type of analysis is more useful than a 
static comparison of an agency's program and performance across 
two points in time because it permits comparison with changing 
national trends. 

At the broadest policy level , these new data are needed to 
address growing demands for accountability in the expenditure of 
public funds. As the competition for limited tax revenues 
becomes more intense, legislative bodies, the media, and tax- 
payers increasingly require that the value of educational and 
other social service expenditures be concretely justified. These 
demands are likely to be particularly insistent in the states 
that have recently enacted educational reform proposals — and 
new r. venue raising and spending plans to implement them. 

These growing requirements for programmatic and fiscal 
accountability provide our backdrop for consideration of how 
these needs can be addressed. 

How to Obtain Better Data on Curriculum and Achievement 

Information on curriculum is cons iderably easier to obtain 
than information on students' mastery of it. In either instance 
it is essential that information requests be limited to the 
minimum needed to yield simple data capable of meeting the needs 
already discussed. For purposes of designing data collection 
procedures, those needs tray be summarized as requirements for 
baseline measures that permit the following: 
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• Comparisons across educational agencies to assess 
relative status 

• Comparisons across time to assess change in educa- 
tional agencies 

• Planning by post-high school providers of education 
and training, including institutions of higher 
educat ion r post secondary vocat ional programs , and 
the Military 

• Assessment by the public to determine if expecta- 
tions for educational delivery and performance are 
being met 

The following discussion focuses first on how NCES could obtain 
data on the content of students' curriculum to address these 
requirements. Then we turn to procedures for obtaining data on 
the extent to which students learn the skills and information 
contained in the curriculum. 



Instructional Content and Methods 

The first step in designing procedures to collect data on 
curricular content is to decide exactly what informction is 
necessary. Possible information categories include for any given 
subject area (e.g., mathematics, language arts): 

• Courses of study offered at each grade, including 
which are reqaired and which are elective 

• Student enrollment in each course 

• Major skills r concepts r and information taught in 
each course 

• Local or state guidelines for skills, concepts, and 
information to be taught in each course 

• Primary teaching techniques used in each course 
(e.g. r laboratory or activity-centered? classroom 
discussion; extens ive use of audio-vi sual materials , 
computers, or self-instructional materials) 
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• Subject-relevant qualifications (e.g. r academic 
preservice training, inservice training, years of 
experience in teaching the course) of teachers 
teaching each course 

• Role of subject supervisors or specialists in 
teaching (or shaping) the course 

These information categories are derived from those used in 
a study conducted by Weiss for the National Science Foundation, 
entitled "1977 National Survey of Science, Mathematics, and 
Social Studies Education." The survey was designed to obtain 
data on course offerings, curriculum usage, and classroom 
practices in science, mathematics, and social studies. Besides 
taking a snapshot of instructional practice in these three 
subjects, NSF used the survey to determine the extent to which 
NSF-sponsored materials were being used in classrooru. and the 
self-reported influence of NSF-sponsored programs of inservice 
training on science, mathematics, and social studies instruction. 
To improve the quality and comprehensiveness of the survey, an 
im^/Ortant design step was the review of the draft questionnaire 
by a large group of consultants with appropriate subject matter 
expertise and representatives of professional associations with 
interests in science, mathematics, and social studies education. 

Another perspective on the collection of instructional data 
is provided by a 1981 survey conducted for the Committee on 
Economic Education of the American Economic Association by 
Yankelovich, Skelly and White, Inc. The purpose of this survey 
was to examine "how economics is being taught in America, i.e., 
who is teaching it, where it is placed in the school curriculum, 
what the focus is, and what teachinc aids and materials economics 
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teachers are using." As in the NSF project, a number of expert 
consultants and association representatives were involved in 
survey design and analysis. This survey collected data in 
categories similar to the seven listed at the beginning of this 
sect ion. 

The econcxnics survey is particularly relevant to the propo- 
sal presented in this paper because it reported information on 
the instructional content of economics courses, using several 
sets of focused subcategories. For example, within the category 
labeled "major skills, concepts, and information taught in each 
course" in the list above, the economics education survey breaks 
out two sets of subcategories. The first is called "goals of 
economic education" and offered respondents the choice among six 
possible goals, including "to help students understand the cur- 
rent problems facing the country" (reported as a very important 
goal by 66 percent of all responding teachers) and "to teach 
students practical skills that they need in their everyday lives, 
such as balancing a checkbook, using credit cards, how to shop 
wisely, etc." (reported as important by 65 percent of the 
respondents). The survey also breaks out a second set of infor- 
mation categories focused on "aspects of economics"; these 
include 23 headings such as "supply and demand" (the most 
frequently taught aspect of economics) and "consumer issues/ 
consumerism" (an aspect taught by 66 percent of all responding 
economics teachers) . 

These two surveys demonstrate the feasibility of obtaining 
nationally representative data on instructional content and 
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methods. They also suggest the desirability of conducting survey 
development separately for each major subject area. This 
strategy would permit the use of expert review panels, such as 
those used in these two surveys. The panels could play valuable 
roles in the review and adoption of subject matter categories 
such as those used in the economics survey. (These categories 
are available in other subject areas and are sometimes called 
"taxonomies of educat ional object ives. " ) 

Students ' Mastery of Instruct ional Content 

The preceding section suggests that any testing of students' 
achievement of curricular goals in a particular subject area 
should not occur until agreement exists on a discrete set of 
objectives or topics for that subject area. At th'^^t point, 
grade-appropriate test items can be developed for each topic or 
objective within a subject area. Obviously, this process will 
need to be as careful and precise as the development of content 
categories, in order for the test items for a particular topic at 
a given grade level to meet criteria such as the following: 

• Accurate measurement across a broad spectrum of dif- 
ficulty levels, in order to determine the level of 
difficulty mastered by a student 

• Assessment across the full breadth of content com- 
monly taught in connnection with a particular objec- 
tive or topic 

• Assessment of differing types of achievement sought 
using a variety of teaching methods 

Although other criteria will be necessary as well, these indicate 
the challenges in designing standard test items tailored to vary- 
ing instructional content and methods. 
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Test administration will require that students be examined 
using the test items that correspond to the objectives or topics 
on which their instiaction has focused. Test results under this 
procedure will be a more accurate measure of the effectiveness of 
instruction than is currently the case with the use of general 
tests of educational achievement. 

Depending on the intended policy uses of the test results, 
tests can be tailored to instructional objectives at any organi- 
zational level desired (e.g., school, district, or state). For 
example, if a state has established improved mathematics compu- 
tation as a major statewide goal, all local agencies may be 
required by the state to test all students in that area. Local 
agencies could then be permitted to administer tests of other 
mathematics objectives (e.g., mathematical reasoning) according 
to local priorities. The ability to tailor such state-level uses 
for the tests will be a major factor in encouraging voluntary 
participation in test development and administration and even 
cost-sharing . 



Problems Likely To Be Encountered in Implementing This New Data 
Collection Focus 

Before implementing this proposal, it will be necessary to 
address and resolve four sets of problems, as described below. 

Problems in reaching agreement among educational interests 
affected by the proposal . This proposal may be seen as intruding 
on state and local prerogatives in the educational enterprise, 
because it would require educational agencies to reveal the 
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priorities they place on different areas of the school curri- 
culum. If widely held, this perception may require that NCES 
distance itself somewhat from the developmental process. One way 
to do this will be to rely heavily on the involvement of (1) 
advisors who are recognized as experts in curricular areas and 
(2) representatives of professional associc ions including both 
those with subject matter orientations and those who represent 
particular parties in the educational process. Using these 
groups and individuals to make qualitative judgments regarding 
the scope and content of necessary surveys and tests is likely to 
reduce public concern with any seemingly inappropriate federal 
involvement. 

Problems in public perception of excessive burden and 
expense . The surveys and testing programs just described will 
inevitably mean new burdens on educational personnel and new 
public expense. The burden and expense could be reduced by use 
of smaller samples, although that strategy will prevent local 
educational agencies and smaller state agencies from obtaining 
information on their educational programs. The ideal arrangement 
would be for states and local educational agencies to find the 
data potentially available from these surveys and tests so desir- 
able that they will use their own resources to pay for them just 
as states are currently able to contract for National Assessment 
for Educational Progress (NAEP) results on a statewide basis. 
Ultimately, however, NCES and ED will simply have to trade off 
the benefits of these survey and test activities against the 
Center's other investments. 
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Problems of encouraging inappropriate comparisons * Like the 
state protests provoked by the Secretary's "Wall Chart/" data 
generated as a result of the proposed surveys and tests will only 
be able to describe a part of the educational program of a pa^^ti* 
cular agency. By omitting such important variables as student 
characteristics, parent and community preferences, and financial 
resources available for education, the new data could generate 
misleading comparisons. Analysis of the new data will thus need 
to consider other, noncurr iculum factors in order to avoid 
erroneous conclusions. 

Problems of encoiupassing diverse educational goals . No 
matter how broad the participation in the developmental process, 
the surveys and tests cannot encompass all of the objectives, 
content, and methods that are used in American elementary and 
secondary education. Because they cannot possibly be perfectly 
comprehensive, educational agencies whose programs "fit" the 
survey structure will ob*^ ain more accurate and useful information 
than will agencies for which the fit is poorer. This problem 
suggests that the developmental process will need to continue 
even after full-scale implementation is under way, in order to 
accommodate state and local diversity and to improve the survey/ 
*-esting fit across agencies. 

Cone Ind i ng Comme n t s 

Because of the potentially large scale of the activities 
required by this proposal, it is clear that NCES could not launch 
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any part of it without extensive advance consultation and 
consensus-building. An early step in this process will be to 
identify actors in the nat ionp.i educational arena that have 
already expressed interest in the?e types of data collection 
activities and to determine what their particular objectives are. 
The Council of Chief State School Officers is one such major 
association and is in fact in a central position to advance and 
even implement parts of this proposal. A second early step will 
be to identify ways of limiting the initial implementation of the 
project. One way would be to confine initial survey and testing 
activities to curriculum and achievement in academic courses at 
the secondary level, since the needs and precedents there seem to 
be clearer than for elementary schooling. Whatever initial 
limits are adopted^ the demand is likely to build for the types 
of information described here. NCES should begin now to consider 
approaches for addressing the demand. 
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